{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "841e533d-ebb3-406d-9da7-b19e2c5f5866",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #04D7FD; padding: 20px; text-align: left;\">\n",
    "    <h1 style=\"color: #000000; font-size: 36px; margin: 0;\">Data Processing for RAG with Data Prep Kit (Python)</h1>\n",
    "    \n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b15976e3",
   "metadata": {},
   "source": [
    "## Before Running the notebook\n",
    "\n",
    "Please complete [setting up python dev environment](./setup-python-dev-env.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "053ecf08-5f62-4b99-9347-8a0955843d21",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This notebook will process PDF documents as part of RAG pipeline\n",
    "\n",
    "![](media/rag-overview-2.png)\n",
    "\n",
    "This notebook will perform steps 1, 2, 3 and 4 in RAG pipeline.\n",
    "\n",
    "Here are the processing steps:\n",
    "\n",
    "- **pdf2parquet** : Extract text (in markdown format) from PDF and store them as parquet files\n",
    "- **Exact Dedup**: Documents with exact content are filtered out\n",
    "- **Chunk documents**: Split the PDFs into 'meaningful sections' (paragraphs, sentences ..etc)\n",
    "- **Text encoder**: Convert chunks into vectors using embedding models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8b10be1",
   "metadata": {},
   "source": [
    "## Step-1: Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "33345487",
   "metadata": {},
   "outputs": [],
   "source": [
    "from my_config import MY_CONFIG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8902eb31",
   "metadata": {},
   "outputs": [],
   "source": [
    "## setup path to utils folder\n",
    "import sys\n",
    "sys.path.append('../utils')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "facb3bbc",
   "metadata": {},
   "source": [
    "## Step-2:  Data\n",
    "\n",
    "We will use white papers  about LLMs.  \n",
    "\n",
    "- [Granite Code Models](https://arxiv.org/abs/2405.04324)\n",
    "- [Attention is all you need](https://arxiv.org/abs/1706.03762)\n",
    "\n",
    "You can of course substite your own data below"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1fe7c0c",
   "metadata": {},
   "source": [
    "### 2.1 - Download data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8739b7a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Cleared input directory\n",
      "\n",
      "input/attention.pdf (2.22 MB) downloaded successfully.\n",
      "\n",
      "input/granite.pdf (1.27 MB) downloaded successfully.\n",
      "\n",
      "input/granite2.pdf (1.27 MB) downloaded successfully.\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "import shutil\n",
    "from file_utils import download_file\n",
    "\n",
    "shutil.rmtree(MY_CONFIG.INPUT_DATA_DIR, ignore_errors=True)\n",
    "shutil.os.makedirs(MY_CONFIG.INPUT_DATA_DIR, exist_ok=True)\n",
    "print (\"✅ Cleared input directory\")\n",
    " \n",
    "download_file (url = 'https://arxiv.org/pdf/1706.03762', local_file = os.path.join(MY_CONFIG.INPUT_DATA_DIR, 'attention.pdf' ))\n",
    "download_file (url = 'https://arxiv.org/pdf/2405.04324', local_file = os.path.join(MY_CONFIG.INPUT_DATA_DIR, 'granite.pdf' ))\n",
    "download_file (url = 'https://arxiv.org/pdf/2405.04324', local_file = os.path.join(MY_CONFIG.INPUT_DATA_DIR, 'granite2.pdf' )) # duplicate\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72510ae6-48b0-4b88-9e13-a623281c3a63",
   "metadata": {},
   "source": [
    "### 2.2 - Set input/output path variables for the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "60ac8bee-0960-4309-b225-d7a211b14262",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Cleared output directory\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "import shutil\n",
    "\n",
    "if not os.path.exists(MY_CONFIG.INPUT_DATA_DIR ):\n",
    "    raise Exception (f\"❌ Input folder MY_CONFIG.INPUT_DATA_DIR = '{MY_CONFIG.INPUT_DATA_DIR}' not found\")\n",
    "\n",
    "output_parquet_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '01_parquet_out')\n",
    "output_exact_dedupe_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '02_dedupe_out')\n",
    "output_chunk_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '03_chunk_out')\n",
    "output_embeddings_dir = os.path.join (MY_CONFIG.OUTPUT_FOLDER, '04_embeddings_out')\n",
    "\n",
    "## clear output folder\n",
    "shutil.rmtree(MY_CONFIG.OUTPUT_FOLDER, ignore_errors=True)\n",
    "shutil.os.makedirs(MY_CONFIG.OUTPUT_FOLDER, exist_ok=True)\n",
    "\n",
    "print (\"✅ Cleared output directory\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2449e5c7-078c-4ad6-a2f6-21d39d4da3fb",
   "metadata": {},
   "source": [
    "## Step-3: docling2parquet -  Convert data from PDF to Parquet\n",
    "\n",
    "This step is reading the input folder containing all PDF files and ingest them in a parquet table using the [Docling package](https://github.com/DS4SD/docling).\n",
    "The documents are converted into a JSON format which allows to easily chunk it in the later steps.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bb15f02-ab5c-4525-a536-cfa1fd2ba70b",
   "metadata": {},
   "source": [
    "### 3.1 - Execute "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4b101999",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-1: Processing input='input' --> output='output/01_parquet_out'\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "16:02:45 INFO - docling2parquet parameters are : {'batch_size': -1, 'artifacts_path': None, 'contents_type': <docling2parquet_contents_types.MARKDOWN: 'text/markdown'>, 'do_table_structure': True, 'do_ocr': True, 'ocr_engine': <docling2parquet_ocr_engine.EASYOCR: 'easyocr'>, 'bitmap_area_threshold': 0.05, 'pdf_backend': <docling2parquet_pdf_backend.DLPARSE_V2: 'dlparse_v2'>, 'double_precision': 8}\n",
      "2025-10-03 16:02:45,521 - INFO - docling2parquet parameters are : {'batch_size': -1, 'artifacts_path': None, 'contents_type': <docling2parquet_contents_types.MARKDOWN: 'text/markdown'>, 'do_table_structure': True, 'do_ocr': True, 'ocr_engine': <docling2parquet_ocr_engine.EASYOCR: 'easyocr'>, 'bitmap_area_threshold': 0.05, 'pdf_backend': <docling2parquet_pdf_backend.DLPARSE_V2: 'dlparse_v2'>, 'double_precision': 8}\n",
      "16:02:45 INFO - pipeline id pipeline_id\n",
      "2025-10-03 16:02:45,522 - INFO - pipeline id pipeline_id\n",
      "16:02:45 INFO - code location {'github': 'UNDEFINED', 'build-date': 'UNDEFINED', 'commit_hash': 'UNDEFINED', 'path': 'UNDEFINED'}\n",
      "2025-10-03 16:02:45,522 - INFO - code location {'github': 'UNDEFINED', 'build-date': 'UNDEFINED', 'commit_hash': 'UNDEFINED', 'path': 'UNDEFINED'}\n",
      "16:02:45 INFO - data factory data_ max_files -1, n_sample -1\n",
      "2025-10-03 16:02:45,522 - INFO - data factory data_ max_files -1, n_sample -1\n",
      "16:02:45 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.pdf'], files to checkpoint ['.parquet']\n",
      "2025-10-03 16:02:45,523 - INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.pdf'], files to checkpoint ['.parquet']\n",
      "16:02:45 INFO - data factory data_ Data Access:  DataAccessLocal\n",
      "2025-10-03 16:02:45,523 - INFO - data factory data_ Data Access:  DataAccessLocal\n",
      "16:02:45 INFO - orchestrator docling2parquet started at 2025-10-03 16:02:45\n",
      "2025-10-03 16:02:45,523 - INFO - orchestrator docling2parquet started at 2025-10-03 16:02:45\n",
      "16:02:45 INFO - Number of files is 3, source profile {'max_file_size': 2.112621307373047, 'min_file_size': 1.2146415710449219, 'total_file_size': 4.541904449462891}\n",
      "2025-10-03 16:02:45,524 - INFO - Number of files is 3, source profile {'max_file_size': 2.112621307373047, 'min_file_size': 1.2146415710449219, 'total_file_size': 4.541904449462891}\n",
      "16:02:45 INFO - Initializing models\n",
      "2025-10-03 16:02:45,525 - INFO - Initializing models\n",
      "2025-10-03 16:02:45,528 - INFO - Initializing pipeline for StandardPdfPipeline with options hash e647edf348883bed75367b22fbe60347\n",
      "2025-10-03 16:02:45,537 - INFO - Loading plugin 'docling_defaults'\n",
      "2025-10-03 16:02:45,538 - INFO - Registered picture descriptions: ['vlm', 'api']\n",
      "2025-10-03 16:02:45,545 - INFO - Loading plugin 'docling_defaults'\n",
      "2025-10-03 16:02:45,547 - INFO - Registered ocr engines: ['easyocr', 'ocrmac', 'rapidocr', 'tesserocr', 'tesseract']\n",
      "2025-10-03 16:02:45,658 - INFO - Accelerator device: 'mps'\n",
      "2025-10-03 16:02:47,789 - INFO - Accelerator device: 'mps'\n",
      "2025-10-03 16:02:48,940 - INFO - Accelerator device: 'mps'\n",
      "2025-10-03 16:02:49,313 - INFO - detected formats: [<InputFormat.PDF: 'pdf'>]\n",
      "2025-10-03 16:02:49,339 - INFO - Going to convert document batch...\n",
      "2025-10-03 16:02:49,340 - INFO - Processing document attention.pdf\n",
      "2025-10-03 16:03:03,432 - INFO - Finished converting document attention.pdf in 14.12 sec.\n",
      "16:03:03 INFO - Completed 1 files (33.33%) in 0.236 min\n",
      "2025-10-03 16:03:03,463 - INFO - Completed 1 files (33.33%) in 0.236 min\n",
      "2025-10-03 16:03:03,464 - INFO - detected formats: [<InputFormat.PDF: 'pdf'>]\n",
      "2025-10-03 16:03:03,466 - INFO - Going to convert document batch...\n",
      "2025-10-03 16:03:03,466 - INFO - Processing document granite.pdf\n",
      "2025-10-03 16:03:51,908 - INFO - Finished converting document granite.pdf in 48.44 sec.\n",
      "16:03:51 INFO - Completed 2 files (66.67%) in 1.044 min\n",
      "2025-10-03 16:03:51,951 - INFO - Completed 2 files (66.67%) in 1.044 min\n",
      "2025-10-03 16:03:51,952 - INFO - detected formats: [<InputFormat.PDF: 'pdf'>]\n",
      "2025-10-03 16:03:51,954 - INFO - Going to convert document batch...\n",
      "2025-10-03 16:03:51,955 - INFO - Processing document granite2.pdf\n",
      "2025-10-03 16:04:42,003 - INFO - Finished converting document granite2.pdf in 50.05 sec.\n",
      "16:04:42 INFO - Completed 3 files (100.0%) in 1.879 min\n",
      "2025-10-03 16:04:42,045 - INFO - Completed 3 files (100.0%) in 1.879 min\n",
      "16:04:42 INFO - Done processing 3 files, waiting for flush() completion.\n",
      "2025-10-03 16:04:42,046 - INFO - Done processing 3 files, waiting for flush() completion.\n",
      "16:04:42 INFO - done flushing in 0.0 sec\n",
      "2025-10-03 16:04:42,046 - INFO - done flushing in 0.0 sec\n",
      "16:04:42 INFO - Completed execution in 1.942 min, execution result 0\n",
      "2025-10-03 16:04:42,053 - INFO - Completed execution in 1.942 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:1 completed successfully\n",
      "CPU times: user 2min 18s, sys: 39.7 s, total: 2min 58s\n",
      "Wall time: 2min 1s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "from dpk_docling2parquet import Docling2Parquet\n",
    "from dpk_docling2parquet import docling2parquet_contents_types\n",
    "\n",
    "STAGE = 1\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{MY_CONFIG.INPUT_DATA_DIR}' --> output='{output_parquet_dir}'\\n\", flush=True)\n",
    "\n",
    "result = Docling2Parquet(input_folder=MY_CONFIG.INPUT_DATA_DIR,\n",
    "                    output_folder=output_parquet_dir,\n",
    "                    data_files_to_use=['.pdf'],\n",
    "                    docling2parquet_contents_type=docling2parquet_contents_types.MARKDOWN,   # markdown\n",
    "                    ).transform()\n",
    "\n",
    "if result == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (f\"❌ Stage:{STAGE}  failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ca790e0",
   "metadata": {},
   "source": [
    "### 3.2 -  Inspect Generated output\n",
    "\n",
    "Here we should see one entry per input file processed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fe59563d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully read 3 parquet files with 3 total rows\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>contents</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>document_id</th>\n",
       "      <th>document_hash</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>document_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>## Granite Code Models: A Family of Open Found...</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>485</td>\n",
       "      <td>f593e73e-b6e4-46e4-a849-fc1222166494</td>\n",
       "      <td>3127757990743433032</td>\n",
       "      <td>pdf</td>\n",
       "      <td>58342470e7d666dca0be87a15fb0552f949a5632606fe1...</td>\n",
       "      <td>121131</td>\n",
       "      <td>2025-10-03T16:03:51.946152</td>\n",
       "      <td>48.444821</td>\n",
       "      <td>granite.pdf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>granite2.pdf</td>\n",
       "      <td>## Granite Code Models: A Family of Open Found...</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>485</td>\n",
       "      <td>dd254934-b0d0-4076-96a5-e14c1830d3d0</td>\n",
       "      <td>3127757990743433032</td>\n",
       "      <td>pdf</td>\n",
       "      <td>58342470e7d666dca0be87a15fb0552f949a5632606fe1...</td>\n",
       "      <td>121131</td>\n",
       "      <td>2025-10-03T16:04:42.039977</td>\n",
       "      <td>50.051021</td>\n",
       "      <td>granite2.pdf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>attention.pdf</td>\n",
       "      <td>Provided proper attribution is provided, Googl...</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>513</td>\n",
       "      <td>a0e5ae6b-cef1-4fab-b63e-c01df20256b2</td>\n",
       "      <td>2949302674760005271</td>\n",
       "      <td>pdf</td>\n",
       "      <td>214960a61e817387f01087f0b0b323cf1ebd8035fffcab...</td>\n",
       "      <td>48981</td>\n",
       "      <td>2025-10-03T16:03:03.458839</td>\n",
       "      <td>14.119857</td>\n",
       "      <td>attention.pdf</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        filename                                           contents  \\\n",
       "0    granite.pdf  ## Granite Code Models: A Family of Open Found...   \n",
       "1   granite2.pdf  ## Granite Code Models: A Family of Open Found...   \n",
       "2  attention.pdf  Provided proper attribution is provided, Googl...   \n",
       "\n",
       "   num_pages  num_tables  num_doc_elements  \\\n",
       "0         28          17               485   \n",
       "1         28          17               485   \n",
       "2         15           4               513   \n",
       "\n",
       "                            document_id        document_hash  ext  \\\n",
       "0  f593e73e-b6e4-46e4-a849-fc1222166494  3127757990743433032  pdf   \n",
       "1  dd254934-b0d0-4076-96a5-e14c1830d3d0  3127757990743433032  pdf   \n",
       "2  a0e5ae6b-cef1-4fab-b63e-c01df20256b2  2949302674760005271  pdf   \n",
       "\n",
       "                                                hash    size  \\\n",
       "0  58342470e7d666dca0be87a15fb0552f949a5632606fe1...  121131   \n",
       "1  58342470e7d666dca0be87a15fb0552f949a5632606fe1...  121131   \n",
       "2  214960a61e817387f01087f0b0b323cf1ebd8035fffcab...   48981   \n",
       "\n",
       "                date_acquired  document_convert_time source_filename  \n",
       "0  2025-10-03T16:03:51.946152              48.444821     granite.pdf  \n",
       "1  2025-10-03T16:04:42.039977              50.051021    granite2.pdf  \n",
       "2  2025-10-03T16:03:03.458839              14.119857   attention.pdf  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from file_utils import read_parquet_files_as_df\n",
    "\n",
    "output_df = read_parquet_files_as_df(output_parquet_dir)\n",
    "\n",
    "# print (\"Output dimensions (rows x columns)= \", output_df.shape)\n",
    "\n",
    "output_df.head(5)\n",
    "\n",
    "## To display certain columns\n",
    "#parquet_df[['column1', 'column2', 'column3']].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f900753",
   "metadata": {},
   "source": [
    "## Step-4: Eliminate Duplicate Documents\n",
    "\n",
    "We have 2 duplicate documnets here : `granite.pdf` and `granite2.pdf`.\n",
    "\n",
    "Note how the `hash` for these documents are same.\n",
    "\n",
    "We are going to perform **de-dupe**\n",
    "\n",
    "On the content of each document, a SHA256 hash is computed, followed by de-duplication of record having identical hashes.\n",
    "\n",
    "[Dedupe transform documentation](https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/ededup/README.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ef93831",
   "metadata": {},
   "source": [
    "### 4.1 - Execute "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1901b4a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-2: Processing input='output/01_parquet_out' --> output='output/02_dedupe_out'\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "16:05:36 INFO - exact dedup params are {'doc_column': 'contents', 'doc_id_column': 'document_id', 'use_snapshot': False, 'snapshot_directory': None}\n",
      "2025-10-03 16:05:36,963 - INFO - exact dedup params are {'doc_column': 'contents', 'doc_id_column': 'document_id', 'use_snapshot': False, 'snapshot_directory': None}\n",
      "16:05:36 INFO - pipeline id pipeline_id\n",
      "2025-10-03 16:05:36,964 - INFO - pipeline id pipeline_id\n",
      "16:05:36 INFO - code location {'github': 'UNDEFINED', 'build-date': 'UNDEFINED', 'commit_hash': 'UNDEFINED', 'path': 'UNDEFINED'}\n",
      "2025-10-03 16:05:36,965 - INFO - code location {'github': 'UNDEFINED', 'build-date': 'UNDEFINED', 'commit_hash': 'UNDEFINED', 'path': 'UNDEFINED'}\n",
      "16:05:36 INFO - data factory data_ max_files -1, n_sample -1\n",
      "2025-10-03 16:05:36,966 - INFO - data factory data_ max_files -1, n_sample -1\n",
      "16:05:36 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "2025-10-03 16:05:36,967 - INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "16:05:36 INFO - data factory data_ Data Access:  DataAccessLocal\n",
      "2025-10-03 16:05:36,968 - INFO - data factory data_ Data Access:  DataAccessLocal\n",
      "16:05:36 INFO - orchestrator ededup started at 2025-10-03 16:05:36\n",
      "2025-10-03 16:05:36,968 - INFO - orchestrator ededup started at 2025-10-03 16:05:36\n",
      "16:05:36 INFO - Number of files is 3, source profile {'max_file_size': 0.04417991638183594, 'min_file_size': 0.020964622497558594, 'total_file_size': 0.10931110382080078}\n",
      "2025-10-03 16:05:36,971 - INFO - Number of files is 3, source profile {'max_file_size': 0.04417991638183594, 'min_file_size': 0.020964622497558594, 'total_file_size': 0.10931110382080078}\n",
      "16:05:36 INFO - Starting from the beginning\n",
      "2025-10-03 16:05:36,972 - INFO - Starting from the beginning\n",
      "16:05:36 INFO - Completed 1 files (33.33%) in 0.0 min\n",
      "2025-10-03 16:05:36,978 - INFO - Completed 1 files (33.33%) in 0.0 min\n",
      "16:05:36 INFO - Completed 2 files (66.67%) in 0.0 min\n",
      "2025-10-03 16:05:36,988 - INFO - Completed 2 files (66.67%) in 0.0 min\n",
      "16:05:36 INFO - Completed 3 files (100.0%) in 0.0 min\n",
      "2025-10-03 16:05:36,996 - INFO - Completed 3 files (100.0%) in 0.0 min\n",
      "16:05:36 INFO - Done processing 3 files, waiting for flush() completion.\n",
      "2025-10-03 16:05:36,996 - INFO - Done processing 3 files, waiting for flush() completion.\n",
      "16:05:36 INFO - done flushing in 0.0 sec\n",
      "2025-10-03 16:05:36,997 - INFO - done flushing in 0.0 sec\n",
      "16:05:36 INFO - Completed execution in 0.001 min, execution result 0\n",
      "2025-10-03 16:05:36,998 - INFO - Completed execution in 0.001 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:2 completed successfully\n",
      "CPU times: user 33.4 ms, sys: 14.5 ms, total: 47.9 ms\n",
      "Wall time: 42.4 ms\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "from dpk_ededup.transform_python import Ededup\n",
    "\n",
    "STAGE = 2\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{output_parquet_dir}' --> output='{output_exact_dedupe_dir}'\\n\", flush=True)\n",
    "\n",
    "result = Ededup(input_folder=output_parquet_dir,\n",
    "    output_folder=output_exact_dedupe_dir,\n",
    "    ededup_doc_column=\"contents\",\n",
    "    ededup_doc_id_column=\"document_id\"\n",
    "    ).transform()\n",
    "\n",
    "if result == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (f\"❌ Stage:{STAGE}  failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c45a59d2",
   "metadata": {},
   "source": [
    "### 4.2 - Inspect Generated output\n",
    "\n",
    "We would see 2 documents: `attention.pdf`  and `granite.pdf`.  The duplicate `granite.pdf` has been filtered out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0691f08e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully read 3 parquet files with 3 total rows\n",
      "Successfully read 2 parquet files with 2 total rows\n",
      "Input files before exact dedupe : 3\n",
      "Output files after exact dedupe : 2\n",
      "Duplicate files removed :   1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>contents</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>document_id</th>\n",
       "      <th>document_hash</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>document_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>attention.pdf</td>\n",
       "      <td>Provided proper attribution is provided, Googl...</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>513</td>\n",
       "      <td>a0e5ae6b-cef1-4fab-b63e-c01df20256b2</td>\n",
       "      <td>2949302674760005271</td>\n",
       "      <td>pdf</td>\n",
       "      <td>214960a61e817387f01087f0b0b323cf1ebd8035fffcab...</td>\n",
       "      <td>48981</td>\n",
       "      <td>2025-10-03T16:03:03.458839</td>\n",
       "      <td>14.119857</td>\n",
       "      <td>attention.pdf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>## Granite Code Models: A Family of Open Found...</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>485</td>\n",
       "      <td>f593e73e-b6e4-46e4-a849-fc1222166494</td>\n",
       "      <td>3127757990743433032</td>\n",
       "      <td>pdf</td>\n",
       "      <td>58342470e7d666dca0be87a15fb0552f949a5632606fe1...</td>\n",
       "      <td>121131</td>\n",
       "      <td>2025-10-03T16:03:51.946152</td>\n",
       "      <td>48.444821</td>\n",
       "      <td>granite.pdf</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        filename                                           contents  \\\n",
       "1  attention.pdf  Provided proper attribution is provided, Googl...   \n",
       "0    granite.pdf  ## Granite Code Models: A Family of Open Found...   \n",
       "\n",
       "   num_pages  num_tables  num_doc_elements  \\\n",
       "1         15           4               513   \n",
       "0         28          17               485   \n",
       "\n",
       "                            document_id        document_hash  ext  \\\n",
       "1  a0e5ae6b-cef1-4fab-b63e-c01df20256b2  2949302674760005271  pdf   \n",
       "0  f593e73e-b6e4-46e4-a849-fc1222166494  3127757990743433032  pdf   \n",
       "\n",
       "                                                hash    size  \\\n",
       "1  214960a61e817387f01087f0b0b323cf1ebd8035fffcab...   48981   \n",
       "0  58342470e7d666dca0be87a15fb0552f949a5632606fe1...  121131   \n",
       "\n",
       "                date_acquired  document_convert_time source_filename  \n",
       "1  2025-10-03T16:03:03.458839              14.119857   attention.pdf  \n",
       "0  2025-10-03T16:03:51.946152              48.444821     granite.pdf  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from file_utils import read_parquet_files_as_df\n",
    "\n",
    "input_df = read_parquet_files_as_df(output_parquet_dir)\n",
    "output_df = read_parquet_files_as_df(output_exact_dedupe_dir)\n",
    "\n",
    "# print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
    "# print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
    "print (f\"Input files before exact dedupe : {input_df.shape[0]:,}\")\n",
    "print (f\"Output files after exact dedupe : {output_df.shape[0]:,}\")\n",
    "print (\"Duplicate files removed :  \", (input_df.shape[0] - output_df.shape[0]))\n",
    "\n",
    "output_df.sample(min(3, output_df.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72274586",
   "metadata": {},
   "source": [
    "##  Step-5: Doc chunks\n",
    "\n",
    "Split the documents in chunks.\n",
    "\n",
    "[Chunking transform documentation](https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/language/doc_chunk/README.md)\n",
    "\n",
    "**Experiment with chunking size to find the setting that works best for your documents**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "369f2cd1",
   "metadata": {},
   "source": [
    "### 5.1 - Execute "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2cfbf532",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-3: Processing input='output/02_dedupe_out' --> output='output/03_chunk_out'\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "16:05:42 INFO - doc_chunk parameters are : {'chunking_type': 'li_markdown', 'content_column_name': 'contents', 'doc_id_column_name': 'document_id', 'output_chunk_column_name': 'contents', 'output_source_doc_id_column_name': 'source_document_id', 'output_jsonpath_column_name': 'doc_jsonpath', 'output_pageno_column_name': 'page_number', 'output_bbox_column_name': 'bbox', 'chunk_size_tokens': 128, 'chunk_overlap_tokens': 30, 'dl_min_chunk_len': None}\n",
      "2025-10-03 16:05:42,332 - INFO - doc_chunk parameters are : {'chunking_type': 'li_markdown', 'content_column_name': 'contents', 'doc_id_column_name': 'document_id', 'output_chunk_column_name': 'contents', 'output_source_doc_id_column_name': 'source_document_id', 'output_jsonpath_column_name': 'doc_jsonpath', 'output_pageno_column_name': 'page_number', 'output_bbox_column_name': 'bbox', 'chunk_size_tokens': 128, 'chunk_overlap_tokens': 30, 'dl_min_chunk_len': None}\n",
      "16:05:42 INFO - pipeline id pipeline_id\n",
      "2025-10-03 16:05:42,333 - INFO - pipeline id pipeline_id\n",
      "16:05:42 INFO - code location {'github': 'UNDEFINED', 'build-date': 'UNDEFINED', 'commit_hash': 'UNDEFINED', 'path': 'UNDEFINED'}\n",
      "2025-10-03 16:05:42,333 - INFO - code location {'github': 'UNDEFINED', 'build-date': 'UNDEFINED', 'commit_hash': 'UNDEFINED', 'path': 'UNDEFINED'}\n",
      "16:05:42 INFO - data factory data_ max_files -1, n_sample -1\n",
      "2025-10-03 16:05:42,333 - INFO - data factory data_ max_files -1, n_sample -1\n",
      "16:05:42 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "2025-10-03 16:05:42,334 - INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "16:05:42 INFO - data factory data_ Data Access:  DataAccessLocal\n",
      "2025-10-03 16:05:42,334 - INFO - data factory data_ Data Access:  DataAccessLocal\n",
      "16:05:42 INFO - orchestrator doc_chunk started at 2025-10-03 16:05:42\n",
      "2025-10-03 16:05:42,334 - INFO - orchestrator doc_chunk started at 2025-10-03 16:05:42\n",
      "16:05:42 INFO - Number of files is 3, source profile {'max_file_size': 0.04416656494140625, 'min_file_size': 0.0028314590454101562, 'total_file_size': 0.067962646484375}\n",
      "2025-10-03 16:05:42,335 - INFO - Number of files is 3, source profile {'max_file_size': 0.04416656494140625, 'min_file_size': 0.0028314590454101562, 'total_file_size': 0.067962646484375}\n",
      "16:05:42 INFO - Completed 1 files (33.33%) in 0.0 min\n",
      "2025-10-03 16:05:42,342 - INFO - Completed 1 files (33.33%) in 0.0 min\n",
      "16:05:42 INFO - Completed 2 files (66.67%) in 0.0 min\n",
      "2025-10-03 16:05:42,350 - INFO - Completed 2 files (66.67%) in 0.0 min\n",
      "16:05:42 WARNING - table is empty, skipping processing\n",
      "2025-10-03 16:05:42,351 - WARNING - table is empty, skipping processing\n",
      "16:05:42 INFO - Completed 3 files (100.0%) in 0.0 min\n",
      "2025-10-03 16:05:42,352 - INFO - Completed 3 files (100.0%) in 0.0 min\n",
      "16:05:42 INFO - Done processing 3 files, waiting for flush() completion.\n",
      "2025-10-03 16:05:42,352 - INFO - Done processing 3 files, waiting for flush() completion.\n",
      "16:05:42 INFO - done flushing in 0.0 sec\n",
      "2025-10-03 16:05:42,353 - INFO - done flushing in 0.0 sec\n",
      "16:05:42 INFO - Completed execution in 0.0 min, execution result 0\n",
      "2025-10-03 16:05:42,353 - INFO - Completed execution in 0.0 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:3 completed successfully\n",
      "CPU times: user 485 ms, sys: 132 ms, total: 617 ms\n",
      "Wall time: 737 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from dpk_doc_chunk.transform_python import DocChunk\n",
    "\n",
    "STAGE = 3\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{output_exact_dedupe_dir}' --> output='{output_chunk_dir}'\\n\", flush=True)\n",
    "\n",
    "result = DocChunk(input_folder=output_exact_dedupe_dir,\n",
    "        output_folder=output_chunk_dir,\n",
    "        doc_chunk_chunking_type= \"li_markdown\",\n",
    "        # doc_chunk_chunking_type= \"dl_json\",\n",
    "        doc_chunk_chunk_size_tokens = 128,  # default 128\n",
    "        doc_chunk_chunk_overlap_tokens=30   # default 30\n",
    "        ).transform()\n",
    "\n",
    "if result == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (f\"❌ Stage:{STAGE}  failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "213afdf6",
   "metadata": {},
   "source": [
    "### 5.2 - Inspect Generated output\n",
    "\n",
    "We would see documents are split into many chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d8138d43",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully read 2 parquet files with 2 total rows\n",
      "Successfully read 2 parquet files with 61 total rows\n",
      "Files processed : 2\n",
      "Chunks created : 61\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>document_hash</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>document_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "      <th>source_document_id</th>\n",
       "      <th>contents</th>\n",
       "      <th>document_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>attention.pdf</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>513</td>\n",
       "      <td>2949302674760005271</td>\n",
       "      <td>pdf</td>\n",
       "      <td>214960a61e817387f01087f0b0b323cf1ebd8035fffcab...</td>\n",
       "      <td>48981</td>\n",
       "      <td>2025-10-03T16:03:03.458839</td>\n",
       "      <td>14.119857</td>\n",
       "      <td>attention.pdf</td>\n",
       "      <td>a0e5ae6b-cef1-4fab-b63e-c01df20256b2</td>\n",
       "      <td>## Attention Visualizations Input-Input Layer5...</td>\n",
       "      <td>8d0f64fb29fb663c3002a06889a79cea125f2e1f86c61a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>attention.pdf</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>513</td>\n",
       "      <td>2949302674760005271</td>\n",
       "      <td>pdf</td>\n",
       "      <td>214960a61e817387f01087f0b0b323cf1ebd8035fffcab...</td>\n",
       "      <td>48981</td>\n",
       "      <td>2025-10-03T16:03:03.458839</td>\n",
       "      <td>14.119857</td>\n",
       "      <td>attention.pdf</td>\n",
       "      <td>a0e5ae6b-cef1-4fab-b63e-c01df20256b2</td>\n",
       "      <td>## 6.2 Model Variations\\n\\nTo evaluate the imp...</td>\n",
       "      <td>af3a7968218bbd719168314e905a7f2e60b1b6ce916192...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>attention.pdf</td>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>513</td>\n",
       "      <td>2949302674760005271</td>\n",
       "      <td>pdf</td>\n",
       "      <td>214960a61e817387f01087f0b0b323cf1ebd8035fffcab...</td>\n",
       "      <td>48981</td>\n",
       "      <td>2025-10-03T16:03:03.458839</td>\n",
       "      <td>14.119857</td>\n",
       "      <td>attention.pdf</td>\n",
       "      <td>a0e5ae6b-cef1-4fab-b63e-c01df20256b2</td>\n",
       "      <td>## Attention Is All You Need\\n\\nAshish Vaswani...</td>\n",
       "      <td>573d1e20f1f71713f4f318a4abdffbb5109437d4fc02a2...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         filename  num_pages  num_tables  num_doc_elements  \\\n",
       "60  attention.pdf         15           4               513   \n",
       "56  attention.pdf         15           4               513   \n",
       "34  attention.pdf         15           4               513   \n",
       "\n",
       "          document_hash  ext  \\\n",
       "60  2949302674760005271  pdf   \n",
       "56  2949302674760005271  pdf   \n",
       "34  2949302674760005271  pdf   \n",
       "\n",
       "                                                 hash   size  \\\n",
       "60  214960a61e817387f01087f0b0b323cf1ebd8035fffcab...  48981   \n",
       "56  214960a61e817387f01087f0b0b323cf1ebd8035fffcab...  48981   \n",
       "34  214960a61e817387f01087f0b0b323cf1ebd8035fffcab...  48981   \n",
       "\n",
       "                 date_acquired  document_convert_time source_filename  \\\n",
       "60  2025-10-03T16:03:03.458839              14.119857   attention.pdf   \n",
       "56  2025-10-03T16:03:03.458839              14.119857   attention.pdf   \n",
       "34  2025-10-03T16:03:03.458839              14.119857   attention.pdf   \n",
       "\n",
       "                      source_document_id  \\\n",
       "60  a0e5ae6b-cef1-4fab-b63e-c01df20256b2   \n",
       "56  a0e5ae6b-cef1-4fab-b63e-c01df20256b2   \n",
       "34  a0e5ae6b-cef1-4fab-b63e-c01df20256b2   \n",
       "\n",
       "                                             contents  \\\n",
       "60  ## Attention Visualizations Input-Input Layer5...   \n",
       "56  ## 6.2 Model Variations\\n\\nTo evaluate the imp...   \n",
       "34  ## Attention Is All You Need\\n\\nAshish Vaswani...   \n",
       "\n",
       "                                          document_id  \n",
       "60  8d0f64fb29fb663c3002a06889a79cea125f2e1f86c61a...  \n",
       "56  af3a7968218bbd719168314e905a7f2e60b1b6ce916192...  \n",
       "34  573d1e20f1f71713f4f318a4abdffbb5109437d4fc02a2...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from file_utils import read_parquet_files_as_df\n",
    "\n",
    "input_df = read_parquet_files_as_df(output_exact_dedupe_dir)  ## for debug purposes\n",
    "output_df = read_parquet_files_as_df(output_chunk_dir)\n",
    "\n",
    "print (f\"Files processed : {input_df.shape[0]:,}\")\n",
    "print (f\"Chunks created : {output_df.shape[0]:,}\")\n",
    "\n",
    "# print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
    "# print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
    "\n",
    "output_df.sample(min(3, output_df.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5370950a-2a3a-4143-8218-f9b4808099ba",
   "metadata": {},
   "source": [
    "## Step-6:   Calculate Embeddings for Chunks\n",
    "\n",
    "we will calculate embeddings for each chunk using an open source embedding model\n",
    "\n",
    "[Embeddings / Text Encoder documentation](https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/language/text_encoder/README.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9112479",
   "metadata": {},
   "source": [
    "### 6.1 - Execute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "23e8b858",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🏃🏼 STAGE-4: Processing input='output/03_chunk_out' --> output='output/04_embeddings_out'\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "16:05:58 INFO - text_encoder parameters are : {'content_column_name': 'contents', 'output_embeddings_column_name': 'embeddings', 'model_name': 'ibm-granite/granite-embedding-30m-english'}\n",
      "2025-10-03 16:05:58,466 - INFO - text_encoder parameters are : {'content_column_name': 'contents', 'output_embeddings_column_name': 'embeddings', 'model_name': 'ibm-granite/granite-embedding-30m-english'}\n",
      "16:05:58 INFO - pipeline id pipeline_id\n",
      "2025-10-03 16:05:58,467 - INFO - pipeline id pipeline_id\n",
      "16:05:58 INFO - code location {'github': 'UNDEFINED', 'build-date': 'UNDEFINED', 'commit_hash': 'UNDEFINED', 'path': 'UNDEFINED'}\n",
      "2025-10-03 16:05:58,468 - INFO - code location {'github': 'UNDEFINED', 'build-date': 'UNDEFINED', 'commit_hash': 'UNDEFINED', 'path': 'UNDEFINED'}\n",
      "16:05:58 INFO - data factory data_ max_files -1, n_sample -1\n",
      "2025-10-03 16:05:58,469 - INFO - data factory data_ max_files -1, n_sample -1\n",
      "16:05:58 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "2025-10-03 16:05:58,469 - INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "16:05:58 INFO - data factory data_ Data Access:  DataAccessLocal\n",
      "2025-10-03 16:05:58,470 - INFO - data factory data_ Data Access:  DataAccessLocal\n",
      "16:05:58 INFO - orchestrator text_encoder started at 2025-10-03 16:05:58\n",
      "2025-10-03 16:05:58,471 - INFO - orchestrator text_encoder started at 2025-10-03 16:05:58\n",
      "16:05:58 INFO - Number of files is 2, source profile {'max_file_size': 0.046176910400390625, 'min_file_size': 0.02873516082763672, 'total_file_size': 0.07491207122802734}\n",
      "2025-10-03 16:05:58,473 - INFO - Number of files is 2, source profile {'max_file_size': 0.046176910400390625, 'min_file_size': 0.02873516082763672, 'total_file_size': 0.07491207122802734}\n",
      "2025-10-03 16:05:58,479 - INFO - Use pytorch device_name: mps\n",
      "2025-10-03 16:05:58,479 - INFO - Load pretrained SentenceTransformer: ibm-granite/granite-embedding-30m-english\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8e266ea3708543fc928d34cea43400cd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5543177a3f43431a9139cd30adcfcaed",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "184afc613fd747d6a5c41fba04b81239",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc9ffb8f666343cfa2cb8a4bc68e8ae6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4510f76cfedc4896863691de95c0ad23",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c206fb061b994987bacc53295828cc45",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "116e1deb0fa041e497ba9b5687fee76f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "85b1b64bdbc44704ad99be5d3fc70de6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e5f21d11c86a40e3b3c40dd6de4157c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "db5518af8c294c3c86bc27f225ad28f6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a122f6aa092241fb9ec0152317979628",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "529b5dd9201c4f2e8adf60fc940efc87",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec2537ddad744a03a912a7b7311b08d2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "24b898fb86f3429b98500f4b94c4a2a9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b5dc771f3b2649c3bceb47775182afab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9e7980f80d4a4cf0bcc9e0a25ab5da5d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "40f1bc1efc734850a733197ee44f761a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "29389310801049d2ae26c65d77dd7655",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1ad58055abf24942980b7d230d249531",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b8c2f4fd82dd4630a31a29e302ddb5bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c0033c8541ee4f159e857a427149c398",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a4ecd01ac8f740b4863bf2b4d5969483",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7e702d993694a06a81c74faf0c73120",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e043599197224f5993ba32bfb9c72e9c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "34c2b804164b4815b41487c86482ad48",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "92d2109a35294390a1347e8471bd2f61",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bbf0fcf6c53842a78f9b1b90c80bc584",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "707745410127442a982eda680b653950",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "16:06:00 INFO - Completed 1 files (50.0%) in 0.008 min\n",
      "2025-10-03 16:06:00,860 - INFO - Completed 1 files (50.0%) in 0.008 min\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a43634135e004e33a279f14eb4246ab9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "047bc1688bff4de7b7346e2c8fe7c40e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2de5171c90c7495ca26167439b50482e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "53d7581e489f4f56b8f73532af9bd64d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d95cdfa9b4d845ef8e9442b65f7a14b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d581cb96c514c20a5e67f05243a0906",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b34f44ebb95843718f44808ddda8bbbb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d872b09e01f34764b0166806faadbd63",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d1b876edab64c2f8a519bdb8106cfc9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ad71c8b5bd244ca4946ad9e0397b466f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "21882a1e9ea34a739bbb058ed9e4256a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3d1889f3e2d14d9192a38c2cdabb3757",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1bad69597f144e729f78902af7efb183",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f27eee11d8df4778ba2bdee6d7ca5ee4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6bdfa520d93b4bcca956357d789308ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66fbd20b121f473885ab60da9e7c424f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b9038b46d3144c268d77192c44b4aff6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba5d28c3a8234a14a1ea48ec4c0efd88",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "25051b1c092f4fbea7476452a6ea4f87",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5e257f7d87054a48810e4a2d30947225",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8bb8bec773c43f3b976f1fdc45b98fe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "152f00d4e4f346f7b68b6c370f3eef26",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6ce3d9a7d7e44455a49684edcd7c2c31",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "457569be33d0400ca5846f8ee9d64303",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f3b379c6123456eaea25377de292721",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d4f5164fc3f45acb0408fb0ad7e35e4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b9d4b81bc3164ad5bd006d7aa36ac380",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "593dd6c672624929921d6a1b0d41dcd6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b725f39f38354004a52f7a135d082ded",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6efb9815db004205a39e77681548c100",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "572791b688cd47d8b1953902626b7e21",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47c6fe2173fd418f845b8ca1a2dd3c74",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6726d3b636cf47dbbcec610018935e3e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "16:06:01 INFO - Completed 2 files (100.0%) in 0.018 min\n",
      "2025-10-03 16:06:01,447 - INFO - Completed 2 files (100.0%) in 0.018 min\n",
      "16:06:01 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "2025-10-03 16:06:01,447 - INFO - Done processing 2 files, waiting for flush() completion.\n",
      "16:06:01 INFO - done flushing in 0.0 sec\n",
      "2025-10-03 16:06:01,448 - INFO - done flushing in 0.0 sec\n",
      "16:06:01 INFO - Completed execution in 0.05 min, execution result 0\n",
      "2025-10-03 16:06:01,449 - INFO - Completed execution in 0.05 min, execution result 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Stage:4 completed successfully\n",
      "CPU times: user 977 ms, sys: 355 ms, total: 1.33 s\n",
      "Wall time: 2.99 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "from dpk_text_encoder.transform_python import TextEncoder\n",
    "\n",
    "STAGE  = 4\n",
    "print (f\"🏃🏼 STAGE-{STAGE}: Processing input='{output_chunk_dir}' --> output='{output_embeddings_dir}'\\n\", flush=True)\n",
    "\n",
    "\n",
    "result = TextEncoder(input_folder= output_chunk_dir, \n",
    "               output_folder= output_embeddings_dir, \n",
    "               text_encoder_model_name = MY_CONFIG.EMBEDDING_MODEL\n",
    "               ).transform()\n",
    "if result == 0:\n",
    "    print (f\"✅ Stage:{STAGE} completed successfully\")\n",
    "else:\n",
    "    raise Exception (f\"❌ Stage:{STAGE}  failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b734852c",
   "metadata": {},
   "source": [
    "### 6.2 - Inspect Generated output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7b1c1d09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully read 2 parquet files with 61 total rows\n",
      "Successfully read 2 parquet files with 61 total rows\n",
      "Input data dimensions (rows x columns)=  (61, 14)\n",
      "Output data dimensions (rows x columns)=  (61, 15)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>num_pages</th>\n",
       "      <th>num_tables</th>\n",
       "      <th>num_doc_elements</th>\n",
       "      <th>document_hash</th>\n",
       "      <th>ext</th>\n",
       "      <th>hash</th>\n",
       "      <th>size</th>\n",
       "      <th>date_acquired</th>\n",
       "      <th>document_convert_time</th>\n",
       "      <th>source_filename</th>\n",
       "      <th>source_document_id</th>\n",
       "      <th>contents</th>\n",
       "      <th>document_id</th>\n",
       "      <th>embeddings</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>485</td>\n",
       "      <td>3127757990743433032</td>\n",
       "      <td>pdf</td>\n",
       "      <td>58342470e7d666dca0be87a15fb0552f949a5632606fe1...</td>\n",
       "      <td>121131</td>\n",
       "      <td>2025-10-03T16:03:51.946152</td>\n",
       "      <td>48.444821</td>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>f593e73e-b6e4-46e4-a849-fc1222166494</td>\n",
       "      <td>## 4 Pretraining\\n\\nIn this section, we provid...</td>\n",
       "      <td>b1dfa24a41fa6d6040704db6bc96c47ac1a09da191af05...</td>\n",
       "      <td>[-0.0498102, -0.06920969, -0.0095467325, 0.026...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>485</td>\n",
       "      <td>3127757990743433032</td>\n",
       "      <td>pdf</td>\n",
       "      <td>58342470e7d666dca0be87a15fb0552f949a5632606fe1...</td>\n",
       "      <td>121131</td>\n",
       "      <td>2025-10-03T16:03:51.946152</td>\n",
       "      <td>48.444821</td>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>f593e73e-b6e4-46e4-a849-fc1222166494</td>\n",
       "      <td>## 2.1 Data Crawling and Filtering\\n\\nThe pret...</td>\n",
       "      <td>64f536c6e279db81ccda1ba14e35ca553e4250de346176...</td>\n",
       "      <td>[0.049523477, -0.015281111, 0.0320079, 0.04259...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>28</td>\n",
       "      <td>17</td>\n",
       "      <td>485</td>\n",
       "      <td>3127757990743433032</td>\n",
       "      <td>pdf</td>\n",
       "      <td>58342470e7d666dca0be87a15fb0552f949a5632606fe1...</td>\n",
       "      <td>121131</td>\n",
       "      <td>2025-10-03T16:03:51.946152</td>\n",
       "      <td>48.444821</td>\n",
       "      <td>granite.pdf</td>\n",
       "      <td>f593e73e-b6e4-46e4-a849-fc1222166494</td>\n",
       "      <td>## 6.1 Code Generation</td>\n",
       "      <td>719ad78af25dc40f13c6d3273faa31d4c5f48b5fbba1fd...</td>\n",
       "      <td>[-0.005764263, 0.014997978, 0.03899995, 0.0295...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       filename  num_pages  num_tables  num_doc_elements        document_hash  \\\n",
       "9   granite.pdf         28          17               485  3127757990743433032   \n",
       "4   granite.pdf         28          17               485  3127757990743433032   \n",
       "16  granite.pdf         28          17               485  3127757990743433032   \n",
       "\n",
       "    ext                                               hash    size  \\\n",
       "9   pdf  58342470e7d666dca0be87a15fb0552f949a5632606fe1...  121131   \n",
       "4   pdf  58342470e7d666dca0be87a15fb0552f949a5632606fe1...  121131   \n",
       "16  pdf  58342470e7d666dca0be87a15fb0552f949a5632606fe1...  121131   \n",
       "\n",
       "                 date_acquired  document_convert_time source_filename  \\\n",
       "9   2025-10-03T16:03:51.946152              48.444821     granite.pdf   \n",
       "4   2025-10-03T16:03:51.946152              48.444821     granite.pdf   \n",
       "16  2025-10-03T16:03:51.946152              48.444821     granite.pdf   \n",
       "\n",
       "                      source_document_id  \\\n",
       "9   f593e73e-b6e4-46e4-a849-fc1222166494   \n",
       "4   f593e73e-b6e4-46e4-a849-fc1222166494   \n",
       "16  f593e73e-b6e4-46e4-a849-fc1222166494   \n",
       "\n",
       "                                             contents  \\\n",
       "9   ## 4 Pretraining\\n\\nIn this section, we provid...   \n",
       "4   ## 2.1 Data Crawling and Filtering\\n\\nThe pret...   \n",
       "16                             ## 6.1 Code Generation   \n",
       "\n",
       "                                          document_id  \\\n",
       "9   b1dfa24a41fa6d6040704db6bc96c47ac1a09da191af05...   \n",
       "4   64f536c6e279db81ccda1ba14e35ca553e4250de346176...   \n",
       "16  719ad78af25dc40f13c6d3273faa31d4c5f48b5fbba1fd...   \n",
       "\n",
       "                                           embeddings  \n",
       "9   [-0.0498102, -0.06920969, -0.0095467325, 0.026...  \n",
       "4   [0.049523477, -0.015281111, 0.0320079, 0.04259...  \n",
       "16  [-0.005764263, 0.014997978, 0.03899995, 0.0295...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from file_utils import read_parquet_files_as_df\n",
    "\n",
    "input_df = read_parquet_files_as_df(output_chunk_dir)\n",
    "output_df = read_parquet_files_as_df(output_embeddings_dir)\n",
    "\n",
    "print (\"Input data dimensions (rows x columns)= \", input_df.shape)\n",
    "print (\"Output data dimensions (rows x columns)= \", output_df.shape)\n",
    "\n",
    "output_df.sample(min(3, output_df.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e12630-be6b-4188-a925-77117155617b",
   "metadata": {},
   "source": [
    "## Step-7: Copy output to final output dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "16dee3b8-31dc-4168-8adb-f2a0a0b5e207",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Copied output from 'output/04_embeddings_out' --> 'output/output_final'\n"
     ]
    }
   ],
   "source": [
    "import shutil\n",
    "\n",
    "shutil.rmtree(MY_CONFIG.OUTPUT_FOLDER_FINAL, ignore_errors=True)\n",
    "shutil.copytree(src=output_embeddings_dir, dst=MY_CONFIG.OUTPUT_FOLDER_FINAL)\n",
    "\n",
    "print (f\"✅ Copied output from '{output_embeddings_dir}' --> '{MY_CONFIG.OUTPUT_FOLDER_FINAL}'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "117b825b-b9cd-4608-a717-34d8abe74edd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
